Incoming population Check

Incoming Tier Wise Distribution

Incoming tier wise distribution : This shows the contribution of each of the tiers( startegic ) for the entire population ( qualified and non qualified ) across different vintages.

Note :

  • Any drastic increase or decrease in contribution of one or more tiers across a set of continous vintages should be flagged.
  • For Qualified accounts vintage considered is Qualified Vintage and for the Babies vintage considered is Bornon Vintage.

Qualified Tier Wise Distribution

Qualified Tier Wise Distribution: This shows the contribution of each of the tiers( startegic ) for the qualified population across different vintages.

Note:

  • Any drastic increase or decrease in Qualified contribution of one or more tiers across a set of continous vintages should be flagged and further invesgated for the reasons.
  • This can be used in conjugation with incoming tier wise distribution to see if there is any change in contibution by one or more tiers from the incoming population to attribute any change.

Tier Qualified Count

Qualified vs Non Qualified Overtime

Qualified vs Non-Qualified overtime: This shows the change in proportion of qualified population across vintages.

Note:

If there is any change in qualification rates across the Vintages this can be due to

  • The type of population we are aquiring ( Can be more riskier or less riskier poulation )
  • Or can be due to relaxing some rules to qualify more customers (This can be invesgated through Tier wise qualification accross different vintages tab )

Qualified to bornon date Difference

Qualified to born on date difference: This divides the qualified population into three sub categories based on the number of days taken for qualification.

  • Category 1: Who are qualified on the same day of born on date
  • Category 2: Who are qualified with in 15 days after born on date
  • Category 3: Who are qualified after 15 days of born on date

Note: If there is no change in the type of population which we are aquiring but if there is increase or decrease in the qualification rate. This can be caused due to people who are not qualified may be getting qualified.

Tier wise qualification accross different vintages

Tier wise qualification across vintages: This shows the proportion of the qualified population with in each tier across different vintages.

Note:

  • Dark Gold signifies a high qualification rate,
  • Light Gold shows a low qualification rate.
  • It is expected to maintain or higher qualification rates in lower risk tiers and lower qualification rate in the higer risk tiers. Any change in any of those qualification proportions needed to flagged.
common_date 0 1 2 3 4 5 6 7 8 9
2018-08-01 NA 0.97 0.96 0.95 0.93 0.89 0.82 0.39 0.22 0.02
2018-09-01 1 0.99 0.98 0.95 0.91 0.88 0.76 0.44 0.23 0.01
2018-10-01 NA 0.95 0.95 0.93 0.92 0.87 0.77 0.53 0.37 0.00
2018-11-01 NA 0.96 0.95 0.93 0.91 0.90 0.81 0.53 0.43 0.00
2018-12-01 NA 0.95 0.96 0.94 0.92 0.89 0.81 0.58 0.41 0.00
2019-01-01 NA 0.95 0.97 0.93 0.92 0.89 0.77 0.56 0.37 NA
2019-02-01 NA 0.96 0.96 0.94 0.91 0.89 0.77 0.61 0.49 0.00
2019-03-01 NA 0.97 0.96 0.92 0.92 0.89 0.78 0.65 0.45 0.01
2019-04-01 NA 0.96 0.94 0.94 0.92 0.91 0.78 0.67 0.46 0.00
2019-05-01 NA 0.96 0.95 0.94 0.92 0.91 0.81 0.62 0.46 NA

PSI Analysis

Definition: Population Stability Index ( PSI ) compares the distribution of output variable (predicted probability) of incoming population to an OOT data set that was used during the model development.

Purpose: Especially in the finance domain, model might be influenced by economic changes. PSI gives an idea about how much the distribution of scores have changed from the time of development. If the distibution has significantly changed over time, the model or strategy should be tweeked/changed accordingly.

Rules:

PSI Value Inference Action
< 0.1 Insignificant Change No action required
> 0.1 and <0.25 Some Minor Change Check other scorecard monitoring metrics
> 0.25 Major shift in population Need to delve deeper

For the calculation of PSI, the incoming population has be divided into tiers on two levels:

  • Strategic tiers : Thresholds assigned on the basis of strategy formed by the underwriting team.
  • Decile tiers : Thresholds assigned on the basic of decile values.

The purpose of generating PSI on two levels is to cross-validate the threshold values. If there is a change in distribution on both levels, we can conclude that there is a change. Otherwise, the threshold values for the tiers should be looked into as psi is sensitive to the bin-size.

The PSI calculation is further divided into based upon:

  • Incoming population : Qualified and non qualified accounts.
  • Qualified population : Only the qualified accounts.

Note:

  • A change in the population distrubution doesn’t mean that model will underperform.We need to closely look into the model performance metrics to arrive some solid conclusion
  • We can also look into the Incoming population Analysis to further enquire about the reasons for the distribution change

Click here for more information about PSI

PSI based on Strategy Tiers

Incoming Population

PSI Summary

Vintage psi_val psi_roll_avg_3 psi_roll_avg_6
Aug-18 0.1 -0.01 -0.01
Sep-18 0.08 -0.01 -0.01
Oct-18 0.06 0.08 -0.01
Nov-18 0.09 0.08 -0.01
Dec-18 0.08 0.08 -0.01
Jan-19 0.07 0.08 0.08
Feb-19 0.05 0.07 0.07
Mar-19 0.07 0.06 0.07
Apr-19 0.08 0.07 0.07
May-19 0.07 0.07 0.07

Tier wise count

Tier wise distribution

Tier wise PSI accross different vintages

Qualified Population

PSI Summary

Vintage psi_val psi_roll_avg_3 psi_roll_avg_6
Aug-18 0.53 -0.01 -0.01
Sep-18 0.64 -0.01 -0.01
Oct-18 0.65 0.61 -0.01
Nov-18 0.69 0.66 -0.01
Dec-18 0.73 0.69 -0.01
Jan-19 1.13 0.85 0.73
Feb-19 0.73 0.86 0.76
Mar-19 0.65 0.84 0.76
Apr-19 0.72 0.7 0.78
May-19 1.11 0.83 0.85

Tier wise count

Tier wise distribution

Tier wise PSI accross different vintages

PSI based on Decile Tiers

Incoming Population

PSI Summary

Vintage psi_val psi_roll_avg_3 psi_roll_avg_6
Aug-18 0.44 -0.01 -0.01
Sep-18 0.39 -0.01 -0.01
Oct-18 0.36 0.4 -0.01
Nov-18 0.36 0.37 -0.01
Dec-18 0.35 0.36 -0.01
Jan-19 0.37 0.36 0.38
Feb-19 0.36 0.36 0.36
Mar-19 0.35 0.36 0.36
Apr-19 0.38 0.36 0.36
May-19 0.36 0.36 0.36

Tier wise count

Tier wise distribution

Tier wise PSI accross different vintages

Qualified Population

PSI Summary

Vintage psi_val psi_roll_avg_3 psi_roll_avg_6
Aug-18 1.38 -0.01 -0.01
Sep-18 1.42 -0.01 -0.01
Oct-18 1.36 1.39 -0.01
Nov-18 1.41 1.4 -0.01
Dec-18 1.42 1.4 -0.01
Jan-19 1.49 1.44 1.41
Feb-19 1.33 1.41 1.4
Mar-19 1.23 1.35 1.37
Apr-19 1.41 1.32 1.38
May-19 1.4 1.35 1.38

Tier wise count

Tier wise distribution

Tier wise PSI accross different vintages

Model Performance

KS and AUC statistics measures the ability of the model to differentiante between events and non events.

Click here for AUC Reference

Click here for KS Reference

AUC KS summarised : This shows a comparison between the expected KS/AUC and the KS/AUC for the simulated OOT sample ascross different vintages.

Note:

  • Any increase in KS/AUC more than 3% is indicated using green
  • Any reduction by 3% specified using red.

For more details regarding the calculation about the AUC and KS refer:

  • Bad_Goods cumilative :
    • Actual Tiers bads-goods cumulative : This shows cummulative bad/goods rates accross different tiers for each Vintage
    • Simulated bad - good cumilative : This shows simulated cummulative bad/goods rates accross different tiers for each Vintage
    • Simulated vs Actual : This shows a detailed comparison on the expected values and the simulated values. This will help out to understand whether it Cummulative bad or good curve effected the AUC/KS values.
  • Within Tier bad rates across vintages : This shows the change in bad rates for tiers over different vintages. This should follow rank ordering. The color coding is done on a tier level.

Note: - Rolling averages can be used to add significance to the trends in the badrates.

AUC-KS Summarised

Roll-Rate-1

AUC

qualify_vintage auc_expected auc_actual
Aug-18 0.64 0.66
Sep-18 0.66 0.67
Oct-18 0.66 0.67
Nov-18 0.68 0.65
Dec-18 0.65 0.59
Jan-19 0.67 0.64
Feb-19 0.67 0.77
Mar-19 0.66 0.58

KS

qualify_vintage ks_expected ks_actual
Aug-18 0.21 0.27
Sep-18 0.23 0.25
Oct-18 0.23 0.29
Nov-18 0.28 0.21
Dec-18 0.23 0.19
Jan-19 0.26 0.21
Feb-19 0.28 0.43
Mar-19 0.25 0.18

Roll-Rate-3

AUC

qualify_vintage auc_expected auc_actual
Oct-18 0.65 0.67
Nov-18 0.67 0.65
Dec-18 0.66 0.59
Jan-19 0.67 0.64
Feb-19 0.66 0.77
Mar-19 0.67 0.58

KS

qualify_vintage ks_expected ks_actual
Oct-18 0.22 0.29
Nov-18 0.25 0.21
Dec-18 0.25 0.19
Jan-19 0.27 0.21
Feb-19 0.25 0.43
Mar-19 0.26 0.18

Roll-Rate-6

AUC

qualify_vintage auc_expected auc_actual
Jan-19 0.66 0.64
Feb-19 0.67 0.77
Mar-19 0.67 0.58

KS

qualify_vintage ks_expected ks_actual
Jan-19 0.25 0.21
Feb-19 0.26 0.43
Mar-19 0.26 0.18

Bads-Goods Cummulative

Actual Tiers bads-goods cummulative

Roll Rate 1

Cum-bads
Cum-goods

Roll Rate 3

Cum-bads
Cum-goods

Roll Rate 6

Cum-bads
Cum-goods

Simulated Tiers bads-goods cummulative

Roll Rate 1

Cum-bads
Cum-goods

#### Roll Rate 3 {.tabset .tabset-dropdown}

Cum-bads
Cum-goods

Roll Rate 6

Cum-bads
Cum-goods

Simulated vs Actual

Roll Rate

Bads Cummulative
Goods Cummulative

Roll Rate 3

Bads Cummulative
Goods Cummulative

Roll Rate 6

Bads Cummulative
Goods Cummulative

Within tier badrates across vintages

Within Tiers Badrates

Rolling Average 1

qualify_vintage 0 1 2 3 4 5 6 7 8 9
2018-08-01 0 0.027 0.069 0.082 0.135 0.173 0.260 0.137 0.226 0.294
2018-09-01 0 0.011 0.036 0.068 0.161 0.049 0.144 0.203 0.198 0.338
2018-10-01 0 0.021 0.039 0.046 0.076 0.097 0.192 0.126 0.141 0.000
2018-11-01 0 0.017 0.039 0.084 0.072 0.114 0.092 0.154 0.182 0.000
2018-12-01 0 0.009 0.080 0.028 0.107 0.085 0.079 0.087 0.101 0.000
2019-01-01 0 0.010 0.041 0.055 0.053 0.084 0.069 0.204 0.057 0.000
2019-02-01 0 0.027 0.038 0.014 0.043 0.060 0.075 0.111 0.347 0.000

Rolling Average 3

qualify_vintage 0 1 2 3 4 5 6 7 8 9
2018-10-01 0 0.021 0.051 0.067 0.125 0.111 0.205 0.147 0.189 0.314
2018-11-01 0 0.017 0.038 0.063 0.106 0.084 0.149 0.153 0.173 0.000
2018-12-01 0 0.017 0.050 0.050 0.085 0.098 0.131 0.126 0.144 0.310
2019-01-01 0 0.012 0.053 0.054 0.080 0.095 0.081 0.148 0.123 0.000
2019-02-01 0 0.014 0.055 0.036 0.076 0.079 0.075 0.134 0.166 0.000

Rolling Average 6

qualify_vintage 0 1 2 3 4 5 6 7 8 9
2019-01-01 0 0.017 0.052 0.063 0.108 0.105 0.161 0.147 0.159 0
2019-02-01 0 0.016 0.045 0.052 0.094 0.082 0.122 0.145 0.170 0

Badrate Within Limit

Rolling Average 1

qualify_vintage 1 2 3 4 5 6 7 8 9
Aug-18 0.015 0.057 0.058 0.104 0.151 0.222 0.069 0.133 0.138
Sep-18 -0.001 0.023 0.044 0.13 0.027 0.106 0.135 0.105 0.182
Oct-18 0.009 0.026 0.022 0.045 0.075 0.154 0.058 0.048 -0.156
Nov-18 0.005 0.026 0.06 0.041 0.092 0.054 0.086 0.089 NA
Dec-18 -0.003 0.068 0.004 0.076 0.063 0.041 0.019 0.008 -0.156
Jan-19 -0.002 0.028 0.031 0.022 0.062 0.031 0.136 -0.036 NA
Feb-19 0.015 0.026 -0.01 0.012 0.038 0.037 0.043 0.254 -0.156

Rolling Average 3

qualify_vintage 1 2 3 4 5 6 7 8 9
Oct-18 0.009 0.038 0.043 0.094 0.089 0.167 0.079 0.096 0.158
Nov-18 0.005 0.026 0.039 0.075 0.062 0.111 0.085 0.08 NA
Dec-18 0.005 0.038 0.026 0.054 0.076 0.093 0.058 0.051 0.154
Jan-19 0 0.04 0.03 0.049 0.073 0.043 0.08 0.03 NA
Feb-19 0.002 0.042 0.012 0.045 0.057 0.037 0.066 0.073 -0.156

Rolling Average 6

qualify_vintage 1 2 3 4 5 6 7 8 9
Jan-19 0.005 0.039 0.039 0.077 0.083 0.123 0.079 0.066 NA
Feb-19 0.004 0.032 0.028 0.063 0.06 0.084 0.077 0.077 NA

Expected Bad Rates

tier bad_rate
1 0.012
2 0.012
3 0.024
4 0.031
5 0.022
6 0.038
7 0.068
8 0.093
9 0.156

Bad-Rate Across different dv

Roll Rate 1

Roll Rate 3

Roll Rate 6

Feature Distribution

CSI Analysis

CSI Summary

Bin Prop

CSI in detail

KS-Statistic